Image Sensor Processing Using a Combined Image and Range Measurement System

ABSTRACT

In one embodiment, an apparatus includes a transmitter operable to transmit a first light beam from a light source. The apparatus also includes a receiver operable to receive a plurality of return light beams and direct the plurality of return light beams through a first beam splitter to an imaging sensor and a LiDAR sensor. The imaging sensor may be operable to process a first portion of the return light beams into image profile data, and the LiDAR sensor may be operable to process a second portion of the return light beams into depth profile data. In addition, the first and second portions of the return light beams may be received from a shared field of view.

BACKGROUND

Light Detection and Ranging (LiDAR) is a sensing method that uses alight beam to measure the distance to various objects. A LiDAR sensorworks by emitting a light beam and measuring the time it takes toreturn. The return time for each return light beam is combined with thelocation of the LiDAR sensor to determine a precise location of asurface point of an object, and this location is recorded as athree-dimensional point in space. An optical camera captures and recordsimages of the external environment. A camera works by opening anaperture to take in light through a lens, and then a light detector(e.g., a charge-coupled device (CCD) or CMOS image sensor) turns thecaptured light into electrical signals including color and brightness ofeach pixel of the image.

Autonomous vehicles typically use a LiDAR sensor to obtain depthprofiles of the environment, and an optical camera to obtain imageprofiles of the environment in order to help navigate the vehicle aroundthe environment. However, because the LiDAR sensor is placed next to aseparate imaging sensor, data from the LiDAR sensor must be calibratedand aligned with the data from the imaging sensor. In addition, problemswith alignment may arise due to the distance between the position of theLiDAR sensor and the position of the imaging sensor and changes inthermal amplitudes over the course of the day in each of the components.Moreover, both components may be prone to drift and calibration errorscaused by vibrations inherent in vehicular movement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an overview of a camera and LiDAR system in anautomotive sensor system.

FIG. 2 illustrates an example schematic of a imaging and rangemeasurement system.

FIG. 3 illustrates example internal components of a imaging and rangemeasurement system.

FIG. 4A illustrates an example of a imaging and range measurement systemwith multiple transmitters for transmitting multiple wavelengths. FIG.4B illustrates an example of a imaging and range measurement system withan optical switch.

FIG. 5 illustrates an example block diagram of a transportationmanagement environment.

FIG. 6 illustrates an example of a computing system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Transportation management technology for “smart” vehicles may be usedfor intelligent transportation and user interaction to help optimizevehicle safety, efficiency, and user-friendliness. A vehicle may usevarious devices and sensors (e.g., LiDAR, cameras, etc.) to sense itsenvironment and navigate around this environment with little to no humaninput. In a regular manually-driven vehicle, these devices and sensormay assist the vehicle operator to more safely or efficiently operatethe vehicle, for example, by using object warning detection indicators,optimizing speed for fuel economy, detecting lane markers and changes,avoiding or minimizing collisions, and/or deploying other relevantvehicle monitoring or maneuvers. In addition, these devices may helptake most or full control of vehicle operation from the vehicle operatorunder some or all circumstances when the transportation managementtechnology is able to recognize a dangerous or risky situation and steeror control the vehicle to avoid or mitigate the situation.

In the case of autonomous vehicles, a vehicle may be equipped with avariety of systems or modules for enabling it to determine itssurroundings and safely and automatically navigate to targetdestinations. For example, an autonomous vehicle may have an integratedcomputing system (e.g., one or more central processing units, graphicalprocessing units, memory, and storage) for controlling variousoperations of the vehicle, such as driving and navigating. To that end,the computing system may process data from one or more sensor arrays.For example, an autonomous vehicle may have optical cameras for, e.g.,recognizing roads and lane markings, and objects on the road; LiDARsfor, e.g., detecting 360° surroundings; infrared cameras for, e.g.,night vision; radio detection and ranging (RADAR) for, e.g., detectingdistant hazards; stereo vision for, e.g., spotting hazards such aspedestrians or tree branches; wheel sensors for, e.g., measuringvelocity; ultra sound for, e.g., parking and obstacle detection; globalpositioning system (GPS) for, e.g., determining the vehicle's currentgeolocation; and/or inertial measurement units, accelerometers,gyroscopes, and/or odometer systems for movement or motion detection.Data from these systems and modules may be used by a navigation systemto safely guide the autonomous vehicle, even without the aid of a humandriver. The autonomous vehicle may also include communication devicesfor, e.g., wirelessly communicating with one or more servers, userdevices (e.g., smartphones, tablet computers, smart wearable devices,laptop computers) and/or other vehicles.

Successful and safe navigation of a vehicle depends on having accuratedata measurement and representation of the external environment at alltimes. In particular embodiments, to maintain an accurate representationor three-dimensional model of the external environment, an opticalcamera may capture a picture of the external environment, and a LiDARinstrument may use a light beam to measure the distance to variousobjects in the external environment. An optical camera works by taking acolor image profile of the environment, and the data collected can beprocessed to read signs, navigate along road markings, recognize movingor stationary objects relevant to the movement of the vehicle, and otherimportant visual driving cues. Multiple optical cameras (e.g., betweenfour to six cameras) may be used to create a three-dimensional image byimage stitching the data from each of the optical cameras. A LiDARinstrument works by emitting a light beam out into the world andmeasuring the time it takes to return to obtain a depth profile of theenvironment. The return time for each return light beam is combined withthe location of the LiDAR instrument to determine a precise location ofa surface point of an object. This location is recorded as athree-dimensional point in space, i.e., azimuth, elevation, and range.In some LiDARs, the Doppler information from the target is acquired,providing a 4D data point. Several recorded three-dimensional points mayprovide an accurate three-dimensional representation of the environmentsurrounding the LiDAR instrument, which may be referred to as a pointcloud. A LiDAR system typically includes a light source, a receiver, amirror that rotates or tilts on a gimbal, timing electronics, a GlobalPositioning System (GPS), and an Inertial Measurement Unit (IMU).

Traditionally, the optical cameras and the LiDAR instrument are separateentities placed in proximity with one another. As such, having accuratedata of a three-dimensional model of the external environment is highlydependent on proper synchronization of the data collected from thevehicle's optical cameras with the data collected from the LiDARinstrument. However, this often requires complex computations inprocessing the camera data and LiDAR data to calibrate and align thereceived image profiles with the respective depth profiles. Moreover,these computations are further complicated by problems in alignmentresulting from changes in thermal amplitudes over the course of the daybetween the two components, the amount of jitter between a camerainterface board and the optical cameras, and drift and calibrationerrors in the LiDAR system caused by vibrations inherent in vehicularmovement.

In particular embodiments, an imaging and range measurement system maybe constructed such that the system uses a common transmitter and acommon receiver and has perfect overlap of the field of view for theimage and depth data collected for the LiDAR sensor and the imagingsensor, in which the imaging sensor (e.g., camera) and LiDAR sensor areone example of the imaging and range measurement system components. Theimaging and range measurement system may have a light source to directlight out of the common transmitter and a common receiver that receivesthe return light beams, which are then directed to a beam splitter thatdirects different wavelengths of light to a LiDAR sensor and a colorsensor within the imaging and range measurement system. In particularembodiments, multiple LiDAR sensors (e.g., TOF CMOS sensor, InGaAssensor) may be used for processing data of different wavelengths. Inaddition, the light source may include an optical switch for switchingbetween different lasers with different wavelengths, which may triggerthe different lasers to transmit light at the same time or at differenttimes.

The imaging and range measurement system may have several advantagesover separate imaging sensors and LiDAR sensors placed next to eachother. The combined camera and LiDAR sensor may help reduce processingpower by maintaining calibration between the LiDAR sensor and theimaging sensor over its lifetime and using the same field of view toreceive both the LiDAR and image data, and thus the resulting depth andimage data is synchronized as collected and no alignment processing isneeded. In addition, because particular embodiments of the combinedcamera and lidar sensor may be a solid-state device with no movingparts, the combined system is athermalized, extremely physically stable,and may be combined into a very small package. Moreover, the benefits ofusing the different wavelengths of light include different penetrationprofiles, less eye safety issues, and longer range (but narrower fieldof view).

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described. In addition, the embodiments disclosedherein are only examples, and the scope of this disclosure is notlimited to them. Particular embodiments may include all, some, or noneof the components, elements, features, functions, operations, or stepsof the embodiments disclosed above. Embodiments according to theinvention are in particular disclosed in the attached claims directed toa method, a storage medium, a system and a computer program product,wherein any feature mentioned in one claim category, e.g., method, canbe claimed in another claim category, e.g., system, as well. Thedependencies or references back in the attached claims are chosen forformal reasons only. However, any subject matter resulting from adeliberate reference back to any previous claims can be claimed as well,so that any combination of claims and the features thereof are disclosedand can be claimed regardless of the dependencies chosen in the attachedclaims. The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

FIG. 1 illustrates an overview of an imaging and range measurementsystem in an automotive sensor system. As illustrated in the example ofFIG. 1, automotive sensor system 100 may include imaging and rangemeasurement systems 102A, 102B, 102C, 102D, 102E, and 102F that arecoupled to a main computer 104 of a vehicle (e.g., a manual-drivenvehicle, an autonomous vehicle, or any other suitable vehicle). Theimaging and range measurement systems 102A-102E may each correspond to afield of vision 106A, 106B, 106C, 106D, 106E, and 106F, respectively, ina ring to capture a 360° view of the environment. In particularembodiments, the number of imaging and range measurement systems may bedifferent from that shown in FIG. 1, such as a configuration with fourimaging and range measurement systems, eight imaging and rangemeasurement systems, or any other suitable number of imaging and rangemeasurement systems. As described in more detail below, each camera andLiDAR sensor system (e.g., 102A) may include an image sensor (e.g., acamera) that is configured to capture individual photo images or aseries of images as a video, and may also include a LiDAR rangemeasurement sensor system that is configured to capture a depth profileof the environment. Although this disclosure describes and illustrates aparticular automotive sensor system having a particular configurationand number of components, this disclosure contemplates any suitableautomotive sensor system having any suitable configuration and number ofcomponents.

FIG. 2 illustrates an example schematic of a imaging and rangemeasurement system. As illustrated in the example of FIG. 2, each of theimaging and range measurement systems 102A-102E may be connected to aninterface 202 through a respective serial link 204. Interface 202 may bemounted inside the vehicle or outside the vehicle (e.g., on the roof)within the sensor array, which is discussed in more detail below.Further, interface 202 may multiplex power, timing, and control datasent to respective imaging and range measurement systems 102A-102E anddata received from the respective imaging and range measurement systems102A-102E connected by its serial link 204. In particular embodiments,interface 202 may include image data processing 206 and depth dataprocessing 208. Image data process 206 may process the image profiledata received from imaging and range measurement systems 102A-102Ereceived via serial link 204. In addition, depth data processing 206 mayprocess the depth image data received from imaging and range measurementsystems 102A-102E received via serial link 204.

In particular embodiments, a timing system 210 coupled to interface 202may provide timing information for operating one or more light sourcesassociated with each of the imaging and range measurement systems 102Athrough serial link 204. Further, timing system 210 is coupled to maincomputer 102 of the vehicle and may provide timestamp information of theimage profile data and depth profile data that is captured by imagingand range measurement systems 102A-102E. In particular embodiments, maincomputer 102 of the vehicle may interface with the outside world andcontrol the overall function of the vehicle. Although this disclosuredescribes and illustrates a schematic of an imaging and rangemeasurement system having a particular configuration of components, thisdisclosure contemplates any suitable imaging and range measurementsystem having any suitable configuration of components.

FIG. 3 illustrates example internal components of an imaging and rangemeasurement system. This system may be used in a manual-driven vehicle,an autonomous vehicle, or any other suitable vehicle for intelligenttransportation management and user interaction to help optimize vehiclesafety, efficiency, and user-friendliness. As illustrated in the exampleof FIG. 3, imaging and range measurement system 102A includes a lightsource 302, which may be an infrared light source, laser light source,or other suitable light source. Light source 302 may direct one or morelight beams 306 through a transmitter 304 to the environment surroundingthe vehicle. As an example and not by way of limitation, light beams 306may have a wavelength between 840 nm to 904 nm. This wavelength rangemay be significant due to the availability of inexpensive lasers andatmospheric windows where humidity and water vapor have littleabsorption. One or more return light beams 308 may be received by areceiver 310 and directed through a beam splitter 312, which may thenseparate received light beams 308 and reflect them to different sensorssimultaneously as a function of wavelength. As an example and not by wayof limitation, beam splitter 312 may be a dichroic mirror. In particularembodiments, received light beam 308 may be separated into a firstportion 316 that is directed to a imaging sensor 314 (e.g., ared-green-blue (RGB) camera sensor) for receiving and processing imageprofile data. As an example and not by way of limitation, first portion316 of received light beams 308 may have a wavelength between 390 nm to780 nm. In addition, received light beam 308 may also be separated intoa second portion 318 that is directed to a LiDAR sensor 320 (e.g., atime-of-flight (TOF) sensor) for receiving and processing depth profiledata. As an example and not by way of limitation, second portion 318 ofreceived light beams 308 may have a wavelength of 780 nm or higher.

In particular embodiments, with regard to the field of view for imagingand range measurement system 120A, transmitter 304 has an associatedfield of view Θ and receiver 310 has an associated field of view Θ′.Field of view Θ for transmitter 304 may completely overlap field of viewΘ′ for receiver 310 such that transmitter 304 and receiver 310 share thesame field of view Θ and Θ′, which are equal to each other andcorrespond to the same field of view, as shown in FIG. 3. In otherwords, received light beams 308 (e.g., the return light beams) that areseparated into first portion 316 and second portion 318 are receivedfrom a “same” or “shared” field of view Θ/Θ′. In addition, light beams306 is directed out into the environment through transmitter 304, andreceived light beams 308 is received through receiver 310 that is commonto both imaging sensor 314 and LiDAR sensor 320. As discussed above,having the same field of view for receiving and processing data using aLiDAR sensor and a imaging sensor with a common transmitter and receiverhas the advantage of allowing the imaging and range measurement systemto collect data that is already synchronized without any additional dataprocessing and thus not requiring any calibration and alignment (e.g.,as would be needed a configuration with separate LiDAR and imagingsensors) due to the perfect overlap of the image and depth profile datareceived by the system. Moreover, additional advantages of the imagingand range measurement system may include being constructed as anathermalized, solid-state device with no moving parts with the benefitof fitting into a very small package.

In particular embodiments, different wavelengths of light may be used inorder to have different penetration profiles, less eye safety issues,and longer image range with a concurrent narrower field of view. FIG. 4Aillustrates an example of an imaging and range measurement system(similar to the system illustrated in FIG. 3) with multiple transmittersfor transmitting multiple wavelengths. FIG. 4B illustrates an example ofan imaging and range measurement system (similar to the systemillustrated in FIG. 3) with an optical switch. As illustrated in theexample of FIG. 4A, imaging and range measurement system 102A includes afirst light source 402 and a second light source 404. Light sources 402,404 may be infrared light sources, laser light sources, other suitablelight sources, or any combination thereof. Light sources 402 and 404 mayeach direct one or more light beams 408 through a transmitter 406 to theenvironment surrounding the vehicle. Light sources 402 and 404 maytransmit light beams 408 at the same time, or at different times (e.g.,with a time period offset as dictated by timing system 210). As anexample and not by way of limitation, first light source 402 maytransmit light beams 408 at a wavelength between 840 nm to 904 nm, andsecond light source 404 may transmit light beams 406 at a wavelength of1550 nm. This wavelength may be significant due to various reasonsincluding telecommunication wavelengths, relatively inexpensive laserand optical components, eye safety, and no absorption by water vapor.

One or more return light beams 410 (e.g., the return light beams) may bereceived by a receiver 412 and directed through a first beam splitter414, which may then separate received light beams 410 and reflect themto different sensors and to another beam splitter simultaneously as afunction of wavelength. As an example and not by way of limitation, beamsplitter 414 may comprise a dichroic mirror. In particular embodiments,received light beam 410 may be separated into a first portion 416 thatis directed to a imaging sensor 418 (e.g., a RGB sensor) for receivingand processing image profile data. As an example and not by way oflimitation, first portion 416 of received light beams 410 may have awavelength between 390 nm to 780 nm. In addition, received light beam410 may also be separated into a second portion 420 that is directed toa second beam splitter 422. This second beam splitter 422 may thenseparate second portion 420 of received light beams 410 into a thirdportion 424 that is directed to a first LiDAR sensor 426 and a fourthportion 428 that is directed to a second LiDAR sensor 430. First andsecond LiDAR sensor 426, 430 maybe each be a TOF sensor, anindium-gallium-arsenide (InGaAs) sensor, or any other sensor suitablefor receiving and processing depth profile data. As an example and notby way of limitation, third portion 424 of received light beams 410 mayhave a wavelength between 840 nm to 904 nm, which may be processed usinga TOF sensor, and fourth portion 428 of received light beams 410 mayhave a wavelength of 1550 nm, which may be processed using an InGaAssensor. In particular embodiments, transmitter 406 has an associatedfield of view Θ″ and receiver 412 has an associated field of view Θ′″.Field of view Θ″ for transmitter 46 may completely overlap field of viewΘ′″ for receiver 412 such that transmitter 46 and receiver 412 share thesame field of view Θ″ and Θ′″, which are equal to each other andcorrespond to the same field of view, as shown in FIG. 4. In otherwords, received light beams 410 that are separated into first portion416, second portion 420, third portion 424, and fourth portion 428 arereceived from a same field of view Θ″/Θ′″. In addition, received lightbeams 410 is received through receiver 412 that is common to imagingsensor 418, first LiDAR sensor 426, and second LiDAR sensor 430.

FIG. 4B illustrates a similar configuration as FIG. 4A with the additionof an optical switch or an optical combiner for more refined control offirst and second light sources 402 and 404. As illustrated in theexample of FIG. 4B, combined first and second light sources 402 and 404are both connected to optical switch 434, which may switch between firstlight source 402 and second light source 404 in order to trigger them atthe same time, or at different times. In particular embodiments, firstlight source 402 and second light source 404 may be triggered totransmit light at different times with a time period offset asdetermined by timing system 210. As an example and not by way oflimitation, the optical switch may be for switching between two laserswith different wavelengths (e.g., 850 nm and 940 nm, 1024 nm and 940 nm,etc.). The wavelength of the lasers may be selected from a range outsideof the 390 nm-780 nm visible range so that the lasers do no interferewith camera function. In particular embodiments, optical switch 434 foruse with multiple light sources may result in the multiple componentsadvantageously sharing as many optical components as possible, and alsoto help with the exact overlapping of the field of view (as discussedabove). Although this disclosure describes and illustrates an imagingand range measurement system having a particular configuration ofcomponents, this disclosure contemplates any suitable imaging and rangemeasurement system having any suitable configuration of components.

FIG. 5 illustrates an example block diagram of a transportationmanagement environment for matching ride requestors with vehicles. Thistransportation management environment may be used in a manual-drivenvehicle, an autonomous vehicle, or any other suitable vehicle forintelligent transportation management and user interaction to helpoptimize vehicle safety, efficiency, and user-friendliness, as discussedabove. In particular embodiments, the vehicle 540 may be an autonomousvehicle and equipped with an array of sensors 544, a navigation system546, and a ride-service computing device 548. In particular embodiments,a fleet of vehicles 540 may be managed by the transportation managementsystem 560. The fleet of vehicles 540, in whole or in part, may be ownedby the entity associated with the transportation management system 560,or they may be owned by a third-party entity relative to thetransportation management system 560. In either case, the transportationmanagement system 560 may control the operations of the vehicles 540,including, e.g., dispatching select vehicles 540 to fulfill riderequests, instructing the vehicles 540 to perform select operations(e.g., head to a service center or charging/fueling station, pull over,stop immediately, self-diagnose, lock/unlock compartments, change musicstation, change temperature, and any other suitable operations), andinstructing the vehicles 540 to enter select operation modes (e.g.,operate normally, drive at a reduced speed, drive under the command ofhuman operators, and any other suitable operational modes).

In particular embodiments, the vehicles 540 may receive data from andtransmit data to the transportation management system 560 and thethird-party system 570. Example of received data may include, e.g.,instructions, new software or software updates, maps, 3-D models,trained or untrained machine-learning models, location information(e.g., location of the ride requestor, the vehicle 540 itself, othervehicles 540, and target destinations such as service centers),navigation information, traffic information, weather information,entertainment content (e.g., music, video, and news) ride requestorinformation, ride information, and any other suitable information.Examples of data transmitted from the vehicle 540 may include, e.g.,telemetry and sensor data, determinations/decisions based on such data,vehicle condition or state (e.g., battery/fuel level, tire and brakeconditions, sensor condition, speed, odometer, etc.), location,navigation data, passenger inputs (e.g., through a user interface in thevehicle 540, passengers may send/receive data to the transportationmanagement system 560 and/or third-party system 570), and any othersuitable data.

In particular embodiments, vehicles 540 may also communicate with eachother as well as other traditional human-driven vehicles, includingthose managed and not managed by the transportation management system560. For example, one vehicle 540 may communicate with another vehicledata regarding their respective location, condition, status, sensorreading, and any other suitable information. In particular embodiments,vehicle-to-vehicle communication may take place over direct short-rangewireless connection (e.g., WI-FI, Bluetooth, NFC) and/or over a network(e.g., the Internet or via the transportation management system 560 orthird-party system 570).

In particular embodiments, a vehicle 540 may obtain and processsensor/telemetry data. Such data may be captured by any suitablesensors. In particular embodiments, vehicle 540 may include opticalcameras that have an image sensor that is configured to captureindividual photo images or a series of images as a video. As an exampleand not by way of limitation, the optical cameras may include acharge-coupled device (CCD) image sensor or a complementarymetal-oxide-semiconductor (CMOS) active-pixel image sensor. Inparticular embodiments, the optical camera may include a lens or lensassembly to collect and focus incoming light onto the focal area of theimage sensor. As an example and not by way of limitation, the opticalcamera may include a fisheye lens, ultra-wide-angle lens, wide-anglelens, or normal lens to focus light onto the image sensor. The opticalcameras may be arranged in a circle or ring that is configured tocapture images over a 360° panoramic view. In particular embodiments,the optical cameras of the vehicle may be organized using apre-determined number (e.g., 6) with overlapping field of views tocapture 3-D visual data. Although this disclosure describes andillustrates particular optical cameras having particular image sensorsand lenses arranged in a particular configuration, this disclosurecontemplates any suitable optical cameras having any suitable imagesensors and lenses arranged in any suitable configuration.

In particular embodiments, the vehicle 540 may have a LiDAR sensor arrayof multiple LiDAR transceivers that are configured to rotate 360°,emitting pulsed laser light and measuring the reflected light fromobjects surrounding vehicle 540. In particular embodiments, LiDARtransmitting signals may be steered by use of a gated light valve, whichmay be a MEMs device that directs a light beam using the principle oflight diffraction. Such a device may not use a gimbaled mirror to steerlight beams in 360° around the vehicle. Rather, the gated light valvemay direct the light beam into one of several optical fibers, which maybe arranged such that the light beam may be directed to many discretepositions around the vehicle. Thus, data may be captured in 360° aroundthe vehicle, but no rotating parts may be necessary. A LiDAR is aneffective sensor for measuring distances to targets, and as such may beused to generate a 3-D model of the external environment of the vehicle540. In particular embodiments, the LiDAR sensor array may include oneor more TOF sensors, one or more InGaAs sensor, any other suitablesensors, or any combination thereof.

In particular embodiments, the 3-D model may represent the externalenvironment including objects such as other cars, curbs, debris,objects, and pedestrians up to a maximum range of the sensor arrangement(e.g., 50, 100, or 200 meters). As an example and not by way oflimitation, the vehicle 540 may have optical cameras pointing indifferent directions. The cameras may be used for, e.g., recognizingroads, lane markings, street signs, traffic lights, police, othervehicles, and any other visible objects of interest. To enable thevehicle 540 to “see” at night, infrared cameras may be installed. Inparticular embodiments, the vehicle may be equipped with stereo visionfor, e.g., spotting hazards such as pedestrians or tree branches on theroad. As another example, the vehicle 540 may have radars for, e.g.,detecting other vehicles and/or hazards afar. Furthermore, the vehicle540 may have ultra sound equipment for, e.g., parking and obstacledetection.

In particular embodiments, in addition to sensors enabling the vehicle540 to detect, measure, and understand the external world around it, thevehicle 540 may further be equipped with sensors for detecting andself-diagnosing the its own state and condition. For example, thevehicle 540 may have wheel sensors for, e.g., measuring velocity; globalpositioning system (GPS) for, e.g., determining the vehicle's currentgeolocation; and/or inertial measurement units, accelerometers,gyroscopes, and/or odometer systems for movement or motion detection.While the description of these sensors provides particular examples ofutility, one of ordinary skill in the art would appreciate that theutilities of the sensors are not limited to those examples. Further,while an example of a utility may be described with respect to aparticular type of sensor, it should be appreciated that the utility maybe achieving using any combination of sensors. For example, a vehicle540 may build a 3D model of its surrounding based on data from itsLiDAR, radar, sonar, and cameras, along with a pre-generated mapobtained from the transportation management system 560 or thethird-party system 570. Although sensors 544 appear in a particularlocation on vehicle 540 in FIG. 5, sensors 544 may be located in anysuitable location in or on vehicle 540. Example locations for sensorsinclude the front and rear bumpers, the doors, the front windshield, onthe side paneling, or any other suitable location.

In particular embodiments, the vehicle 540 may be equipped with aprocessing unit (e.g., one or more CPUs and GPUs), memory, and storage.The vehicle 540 may thus be equipped to perform a variety ofcomputational and processing tasks, including processing the sensordata, extracting useful information, and operating accordingly. Forexample, based on images captured by its optical cameras and amachine-vision model, the vehicle 540 may identify particular types ofobjects captured by the images, such as pedestrians, other vehicles,lanes, curbs, and any other objects of interest.

In particular embodiments, processing unit associated with vehicle 540may receive autonomous-vehicle sensor data that represents an externalenvironment within a threshold distance of vehicle 540. In particularembodiments, the computing device may be a ride-service computingdevice, navigation system, or may be any other suitable computing deviceassociated with vehicle 540. The autonomous-vehicle sensor data may becollected via sensors arranged on the outside or the inside of vehicle540. The autonomous-vehicle sensor data may enable vehicle 540 toidentify objects in the surrounding external environment, such as othervehicles, obstacles, traffic signage, cyclists, or pedestrians.

In particular embodiments, the autonomous-vehicle sensor data mayrepresent a three-dimensional schema of the external environment ofvehicle 540. As an example and not by way of limitation, thethree-dimensional schema may represent the external environmentincluding objects such as for example other cars and pedestrians up to amaximum range of the sensor array 144 (e.g. 100 meters). In particularembodiments, at least some of the autonomous-vehicle sensor data may belabeled to include references to objects that are within a thresholddistance from vehicle 540. The autonomous-vehicle sensor data mayfurther enable vehicle 540 to identify the road upon which it isdriving, lanes in the road, or any other suitable object.

In particular embodiments, vehicle 540 may combine theautonomous-vehicle sensor data from multiple types of sensors with othertypes of data to detect roadways, buildings, traffic signs, and otherobjects. The other types of data may include data acquired from thirdparties. Examples of other types of data acquired from third partiesinclude map data, traffic data, weather data, ratings data (e.g. from anonline review website or another third-party ratings entity) or anyother suitable type of data. Although this disclosure describesreceiving sensor data in a particular manner, this disclosurecontemplates receiving sensor data in any suitable manner.

In particular embodiments, the vehicle 540 may have a navigation system546 responsible for safely navigating the vehicle 540. In particularembodiments, the navigation system 546 may take as input any type ofsensor data from, e.g., a Global Positioning System (GPS) module,inertial measurement unit (IMU), LiDAR transceivers, optical cameras,radio frequency (RF) transceivers, or any other suitable telemetry orsensory mechanisms. The navigation system 546 may also utilize, e.g.,map data, traffic data, accident reports, weather reports, instructions,target destinations, and any other suitable information to determinenavigation routes and particular driving operations (e.g., slowing down,speeding up, stopping, swerving, etc.). In particular embodiments, thenavigation system 546 may use its determinations to control the vehicle540 to operate in prescribed manners and to guide the vehicle 540 to itsdestinations without colliding into other objects. Although the physicalembodiment of the navigation system 546 (e.g., the processing unit)appears in a particular location on vehicle 540 in FIG. 5, navigationsystem 546 may be located in any suitable location in or on vehicle 540.Example locations for navigation system 546 include inside the cabin orpassenger compartment of vehicle 540, near the engine/battery, near thefront seats, rear seats, or in any other suitable location. A vehicle540 may include one or more sensors of various types in a sensor arrayto capture information of the external environment of vehicle 540.Although sensor array 544 is illustrated in a particular location onvehicle 540 in FIG. 5, sensor array 544 may be located in any suitablelocation in or on vehicle 540. Example locations for sensors include thefront and rear bumpers, the doors, the front windshield, on the sidepaneling, or any other suitable location of vehicle 540. In particularembodiments, a navigation system of vehicle 540 may be any suitableautonomous navigation system, such as for example a navigation systembased at least in part on a Global Positioning System (GPS) module,inertial measurement unit (IMU), light detection and ranging (LiDAR)transceivers, optical cameras, radio-frequency (RF) transceivers,ultrasonic sensors, or any other suitable data gathering mechanism.While vehicle 540 is being operated, vehicle 540 may share data (e.g.sensor data, navigation data) with a ride-service system.Autonomous-vehicle sensor data may be captured by any suitable sensorarrangement or array.

FIG. 6 illustrates an example computer system. In particularembodiments, one or more computer systems 600 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 600 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 600 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 600.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems600. This disclosure contemplates computer system 600 taking anysuitable physical form. As example and not by way of limitation,computer system 600 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a laptop or notebook computer system, an interactive kiosk, amainframe, a mesh of computer systems, a mobile telephone, a personaldigital assistant (PDA), a server, a tablet computer system, anaugmented/virtual reality device, or a combination of two or more ofthese. Where appropriate, computer system 600 may include one or morecomputer systems 600; be unitary or distributed; span multiplelocations; span multiple machines; span multiple data centers; or residein a cloud, which may include one or more cloud components in one ormore networks. Where appropriate, one or more computer systems 600 mayperform without substantial spatial or temporal limitation one or moresteps of one or more methods described or illustrated herein. As anexample and not by way of limitation, one or more computer systems 600may perform in real time or in batch mode one or more steps of one ormore methods described or illustrated herein. One or more computersystems 600 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate.

In particular embodiments, computer system 600 includes a processor 602,memory 604, storage 606, an input/output (I/O) interface 608, acommunication interface 610, and a bus 612. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 602 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 602 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 604, or storage 606; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 604, or storage 606. In particular embodiments, processor602 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 602 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 602 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 604 or storage 606, andthe instruction caches may speed up retrieval of those instructions byprocessor 602. Data in the data caches may be copies of data in memory604 or storage 606 for instructions executing at processor 602 tooperate on; the results of previous instructions executed at processor602 for access by subsequent instructions executing at processor 602 orfor writing to memory 604 or storage 606; or other suitable data. Thedata caches may speed up read or write operations by processor 602. TheTLBs may speed up virtual-address translation for processor 602. Inparticular embodiments, processor 602 may include one or more internalregisters for data, instructions, or addresses. This disclosurecontemplates processor 602 including any suitable number of any suitableinternal registers, where appropriate. Where appropriate, processor 602may include one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 602. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 604 includes main memory for storinginstructions for processor 602 to execute or data for processor 602 tooperate on. As an example and not by way of limitation, computer system600 may load instructions from storage 606 or another source (such as,for example, another computer system 600) to memory 604. Processor 602may then load the instructions from memory 604 to an internal registeror internal cache. To execute the instructions, processor 602 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 602 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor602 may then write one or more of those results to memory 604. Inparticular embodiments, processor 602 executes only instructions in oneor more internal registers or internal caches or in memory 604 (asopposed to storage 606 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 604 (as opposedto storage 606 or elsewhere). One or more memory buses (which may eachinclude an address bus and a data bus) may couple processor 602 tomemory 604. Bus 612 may include one or more memory buses, as describedin further detail below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 602 and memory 604 andfacilitate accesses to memory 604 requested by processor 602. Inparticular embodiments, memory 604 includes random access memory (RAM).This RAM may be volatile memory, where appropriate. Where appropriate,this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 604 may include one ormore memories 604, where appropriate. Although this disclosure describesand illustrates particular memory, this disclosure contemplates anysuitable memory.

In particular embodiments, storage 606 includes mass storage for data orinstructions. As an example and not by way of limitation, storage 606may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage606 may include removable or non-removable (or fixed) media, whereappropriate. Storage 606 may be internal or external to computer system600, where appropriate. In particular embodiments, storage 606 isnon-volatile, solid-state memory. In particular embodiments, storage 606includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 606 taking any suitable physicalform. Storage 606 may include one or more storage control unitsfacilitating communication between processor 602 and storage 606, whereappropriate. Where appropriate, storage 606 may include one or morestorages 606. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 608 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 600 and one or more I/O devices. Computer system600 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 600. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 608 for them. Where appropriate, I/O interface 608 mayinclude one or more device or software drivers enabling processor 602 todrive one or more of these I/O devices. I/O interface 608 may includeone or more I/O interfaces 608, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 610 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 600 and one or more other computer systems 600 or one ormore networks. As an example and not by way of limitation, communicationinterface 610 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 610 for it. As an example and not by way of limitation,computer system 600 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 600 may communicate with a wireless PAN (WPAN)(such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 600 may include any suitable communication interface 610 for anyof these networks, where appropriate. Communication interface 610 mayinclude one or more communication interfaces 610, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

In particular embodiments, bus 612 includes hardware, software, or bothcoupling components of computer system 600 to each other. As an exampleand not by way of limitation, bus 612 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 612may include one or more buses 612, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

1. An apparatus comprising: a transmitter operable to transmit a firstlight beam from a light source; and a receiver operable to: receive aplurality of return light beams, and direct the plurality of returnlight beams through a first beam splitter to an imaging sensor and aLiDAR sensor, the imaging sensor being operable to process a firstportion of the return light beams into image profile data, and the LiDARsensor being operable to process a second portion of the return lightbeams into depth profile data, wherein the first and second portions ofthe return light beams are received from a shared field of view.
 2. Theapparatus of claim 1, wherein the first portion of the return lightbeams is within a different wavelength range from the second portion ofthe return light beams.
 3. The apparatus of claim 1, wherein thereceiver is further operable to: direct the return light beams through asecond beam splitter to a second LiDAR sensor operable to process athird portion of the return light beams.
 4. The apparatus of claim 3,wherein the second LiDAR sensor is different from the LiDAR sensor. 5.The apparatus of claim 3, wherein the third portion of the return lightbeams is different from the first and second portions of the returnlight beams.
 6. The apparatus of claim 3, wherein the first beamsplitter and the second beam splitter each comprise a dichroic mirror,and wherein the imaging sensor comprises a camera sensor.
 7. Theapparatus of claim 3, wherein the first, second, and third portions ofthe return light beams are received from the shared field of view. 8.The apparatus of claim 1, further comprising an optical switch operableto switch between sending the first light beam from the light source anda second light beam from a second light source to be transmitted by thetransmitter.
 9. The apparatus of claim 8, wherein the first light beamis a different wavelength from the second light beam.
 10. A methodcomprising: transmitting, via a transmitter, a first light beam from alight source; receiving, via a receiver, a plurality of return lightbeams; and directing, via a first beam splitter, the plurality of returnlight beams to an imaging sensor and a LiDAR sensor, the imaging sensorbeing operable to process a first portion of the return light beams intoimage profile data, and the LiDAR sensor being operable to process asecond portion of the return light beams into depth profile data,wherein the first and second portions of the return light beams arereceived from a shared field of view.
 11. The method of claim 10,wherein the first portion of the return light beams is within adifferent wavelength range from the second portion of the return lightbeams.
 12. The method of claim 10, further comprising: directing, via asecond beam splitter, the return light beams to a second LiDAR sensoroperable to process a third portion of the return light beams.
 13. Themethod of claim 12, wherein the second LiDAR sensor is different fromthe LiDAR sensor.
 14. The method of claim 12, wherein the third portionof the return light beams is different from the first and secondportions of the return light beams.
 15. The method of claim 12, whereinthe first beam splitter and the second beam splitter each comprise adichroic mirror, and wherein the imaging sensor comprises a camerasensor.
 16. The method of claim 12, wherein the first, second, and thirdportions of the return light beams are received from the shared field ofview.
 17. The method of claim 10, further comprising: switching, via anoptical switch, between sending the first light beam from the lightsource and a second light beam from a second light source to betransmitted by the transmitter.
 18. The method of claim 17, wherein thefirst light beam is a different wavelength from the second light beam.19. An apparatus comprising: means for transmitting a first light beamfrom a light source; means for receiving a plurality of return lightbeams; and means for directing the plurality of return light beams to animaging sensor and a LiDAR sensor, the imaging sensor being operable toprocess a first portion of the return light beams into image profiledata, and the LiDAR sensor being operable to process a second portion ofthe return light beams into depth profile data, wherein the first andsecond portions of the return light beams are received from a sharedfield of view.
 20. The apparatus of claim 1, wherein the second portionof the return light beams corresponds to a reflection of the first lightbeam.
 21. The apparatus of claim 1, wherein the first and secondportions of the return light beams received from the shared field ofview originate from a same portion of an object.
 22. The apparatus ofclaim 1, wherein the image profile data and the depth profile data aresubstantially synchronized.